60 research outputs found
Stochastic IMT (insulator-metal-transition) neurons: An interplay of thermal and threshold noise at bifurcation
Artificial neural networks can harness stochasticity in multiple ways to
enable a vast class of computationally powerful models. Electronic
implementation of such stochastic networks is currently limited to addition of
algorithmic noise to digital machines which is inherently inefficient; albeit
recent efforts to harness physical noise in devices for stochasticity have
shown promise. To succeed in fabricating electronic neuromorphic networks we
need experimental evidence of devices with measurable and controllable
stochasticity which is complemented with the development of reliable
statistical models of such observed stochasticity. Current research literature
has sparse evidence of the former and a complete lack of the latter. This
motivates the current article where we demonstrate a stochastic neuron using an
insulator-metal-transition (IMT) device, based on electrically induced
phase-transition, in series with a tunable resistance. We show that an IMT
neuron has dynamics similar to a piecewise linear FitzHugh-Nagumo (FHN) neuron
and incorporates all characteristics of a spiking neuron in the device
phenomena. We experimentally demonstrate spontaneous stochastic spiking along
with electrically controllable firing probabilities using Vanadium Dioxide
(VO) based IMT neurons which show a sigmoid-like transfer function. The
stochastic spiking is explained by two noise sources - thermal noise and
threshold fluctuations, which act as precursors of bifurcation. As such, the
IMT neuron is modeled as an Ornstein-Uhlenbeck (OU) process with a fluctuating
boundary resulting in transfer curves that closely match experiments. As one of
the first comprehensive studies of a stochastic neuron hardware and its
statistical properties, this article would enable efficient implementation of a
large class of neuro-mimetic networks and algorithms.Comment: Added sectioning, Figure 6, Table 1, and Section II.E Updated
abstract, discussion and corrected typo
Direct Feedback Alignment with Sparse Connections for Local Learning
Recent advances in deep neural networks (DNNs) owe their success to training
algorithms that use backpropagation and gradient-descent. Backpropagation,
while highly effective on von Neumann architectures, becomes inefficient when
scaling to large networks. Commonly referred to as the weight transport
problem, each neuron's dependence on the weights and errors located deeper in
the network require exhaustive data movement which presents a key problem in
enhancing the performance and energy-efficiency of machine-learning hardware.
In this work, we propose a bio-plausible alternative to backpropagation drawing
from advances in feedback alignment algorithms in which the error computation
at a single synapse reduces to the product of three scalar values. Using a
sparse feedback matrix, we show that a neuron needs only a fraction of the
information previously used by the feedback alignment algorithms. Consequently,
memory and compute can be partitioned and distributed whichever way produces
the most efficient forward pass so long as a single error can be delivered to
each neuron. Our results show orders of magnitude improvement in data movement
and improvement in multiply-and-accumulate operations over
backpropagation. Like previous work, we observe that any variant of feedback
alignment suffers significant losses in classification accuracy on deep
convolutional neural networks. By transferring trained convolutional layers and
training the fully connected layers using direct feedback alignment, we
demonstrate that direct feedback alignment can obtain results competitive with
backpropagation. Furthermore, we observe that using an extremely sparse
feedback matrix, rather than a dense one, results in a small accuracy drop
while yielding hardware advantages. All the code and results are available
under https://github.com/bcrafton/ssdfa.Comment: 15 pages, 8 figure
Fusing Frame and Event Vision for High-speed Optical Flow for Edge Application
Optical flow computation with frame-based cameras provides high accuracy but
the speed is limited either by the model size of the algorithm or by the frame
rate of the camera. This makes it inadequate for high-speed applications. Event
cameras provide continuous asynchronous event streams overcoming the frame-rate
limitation. However, the algorithms for processing the data either borrow frame
like setup limiting the speed or suffer from lower accuracy. We fuse the
complementary accuracy and speed advantages of the frame and event-based
pipelines to provide high-speed optical flow while maintaining a low error
rate. Our bio-mimetic network is validated with the MVSEC dataset showing 19%
error degradation at 4x speed up. We then demonstrate the system with a
high-speed drone flight scenario where a high-speed event camera computes the
flow even before the optical camera sees the drone making it suited for
applications like tracking and segmentation. This work shows the fundamental
trade-offs in frame-based processing may be overcome by fusing data from other
modalities
- …